feat(pr-workflow): add the /mms-evidence orchestrator, runners, and publish gate - #84

Two things in this PR shared a root and a two-arm shape while answering different
questions: the skill compares a base commit against a branch to show a test is
connected to the reported bug, and `falsify-probe.sh` mutates one line at one
commit to show a suite notices the mechanism going away. A test can satisfy
either and fail the other, so the names had to stop rhyming.

`red-on-base` names the skill's own discipline — the test must be red on the base
commit — and shares no root with the runner. The evidence category keeps its name:
a falsifying regression test is the artifact, and this is the procedure that
produces one.

The runner's stderr described its own result as "a falsifying test", which was the
same collision inside the output of the thing causing it; it now says what it
proves and points at the other experiment by name.

Every other engine `evidence` names ships in its own pull request —
`memory-leak`, `race-condition-repro`, `supply-chain-audit`,
`react-render-delta`, `agent-run-cost`. This one was the sole exception, and the
reason was authoring order rather than design: it was written alongside the
orchestrator before the split between the instrument and the reasoning that
points it existed as a concept.

It lands in #106 with the other reasoning skills, whose substance is the same
kind — what counts as proof, and how a proof can look right while testing the
wrong thing. What stays here is the machinery: the runners, the run workflow, the
gate, the hooks.

The B3 engine cell now names a skill that arrives in #106, which is a dangling
name in a table rather than a broken link, and resolves whichever order the two
merge.

Eleven checks tested properties of the comment text. Text can be perfect and land
somewhere nobody will read it, and that is what happened: across one register of
published runs, 22 of 27 comments went onto pull requests that had already merged
when they were posted — median 22 days after the merge, one 178 days after. The
gate passed every one, because no property of a comment reveals the state of its
destination.

Check 12 takes `--target owner/repo#N` and blocks on anything that is not open.
Omitting the target fails rather than passes: an unchecked destination is the
condition that produced all 22. Without `gh` on PATH it reports UNVERIFIED and
still refuses, since the point is that silence here is indistinguishable from
success.

The CI workflow sources runners by `skills_ref`, and every run this week pointed
at a fork branch. Four fixes were made there and never reached this one: the
mutation now travels through `ENVIRON` instead of an escape-processed `awk -v`
assignment, the artifact reports the line read back off disk beside the line
requested, `--expect-fail` turns a red arm in the wrong place into its own
outcome, and `--metric` lets the caller name what a probe counted.

So the defect this PR's own description cites as the reason instruments must
report their effect was, until now, still live in the instrument this PR ships.
The probe comes across too, with the import fix that made it resolve at the
destination the workflow copies it to.

Two copies of the same scripts on two branches, edited in both directions —
`attest-gate.sh` had a check the fork lacked, so it stays as it is here.

Check 9 is called "verdict matches artifact" and compares verdict words. Nothing
compared the numbers, and prose drifting from the exhibit beside it is the most
common way one of these goes wrong.

Found by building a demonstration artifact to test this gate: the prose read
"0 errors over 48 skills" directly above an exhibit reading "47 skill(s)
checked", and named a warning class with zero instances in the output it was
describing. Every other check passed. Two independent readers caught both, which
is the argument for moving it into the layer that always runs rather than the one
that costs money and sometimes never reports.

Narrow on purpose, because a noisy check is an ignored one. Two-plus digits only,
and only those absent from every fenced block; whole URLs, issue refs, versions,
dates, SHAs, file:line citations, hyphenated identifiers and regex quantifiers
are excluded as references rather than measurements. Each exclusion was added
after a control run flagged something that was not a figure.

Across eight real artifacts it flags one, correctly: a verdict line quoting an
author's "730 tests" beside an exhibit measuring 731, where nothing distinguishes
the cited figure from the measured one.

With `gh` absent the check printed UNVERIFIED and exited 0, so on a machine
without it — running locally — the destination check announced that it had not
run and the gate reported clean. A control that cannot run is indistinguishable
from one that passed unless the exit code says otherwise.

Renamed on the security-domain branch; installs as `mms-lavamoat-policy`.

A diligence comment renders no verdict and deliberately does not use the Validation Run
envelope. That exemption meant it was checked by nothing: this gate only knew the
Validation Run shape, and `pr-evidence-gate.py` by design does not trip on a body
claiming no verdict. So every rule the diligence skills state about their own output —
including "runtime claims need a runtime artifact" — had no execution path.

It showed. A lavamoat comment shipped with no marker pair, an `npm pack` specifier set
no reader could fetch, and two bare integers traceable to nothing.

`--diligence` swaps the four envelope checks for that contract's own — its marker pair,
its header, permalinks pinned to a tag or SHA rather than a branch head, and a runtime
claim check asking for the thing a `/blob/` link cannot witness. 3, 8 and 9 report SKIP
with the reason rather than passing silently, since a check that cannot fail should not
read as a check that passed. Everything downstream of the envelope is shared, because
those defects are shared.

Run against the comment that prompted this, it fails 1, 5 and 13 and passes the rest.

This repository is public. Naming a private repository here discloses its existence,
its owner and roughly its contents to every reader — and a prohibition naming it
("do not re-host to X, it is private") discloses exactly as much as a recommendation
would. Four such references were doing that, and the guidance survives without them:
the rule is audience-reachability, which is stated directly rather than by example.

Two memory-file citations offered as "source of truth" pointed into a private repo, so
a reader was told to follow a rule whose justification they cannot open. The reasoning
is inlined; the pointer is gone.

The publish-surface snippet hardcoded a GitHub username, which decided the destination
for whoever ran it. Now derived from `gh api user --jq .login`, and the surrounding
prose is second-person rather than first — a shared skill has no "my PRs".

`/attest` is no longer linked to a personal repository. That leaves it named but not
resolvable, which is honest and is the smaller problem; the workflow depending on a
command nobody else has is tracked separately.

The re-hosting section named a personal S3 bucket, its region, its prefix layout, and
stated that anonymous `GetObject` is enabled under `public/*`. That is a live
unauthenticated endpoint advertised, with its structure, to every reader of a public
repository — a larger disclosure than the repository names removed alongside it, and
one that reads as configuration rather than as a secret, which is why it survived two
passes over this file.

Now `EVIDENCE_BUCKET` and `EVIDENCE_REGION` from the environment. The requirements the
bucket must satisfy — anonymous GetObject under `public/*`, listing disabled — are
stated, because those are the load-bearing part; the name never was.

The G5 lane likewise named a private test fork, which carried both the org and a
personal handle. Now "your own test fork".

…setup

Removing the hardcoded username left `gh pr view --jq --arg me "$ME"`, which is not a
thing gh supports — its built-in filter takes no --arg and the command dies with
"accepts at most 1 arg(s)". Piped to real jq instead, and checked against both branches
of the logic: a PR authored by someone else resolves to "skip", one authored by the
caller to "body".

Replacing the named bucket with `EVIDENCE_BUCKET` removed a working default and put
nothing in its place, so the section told you to configure a bucket without saying what
"conforming" meant. The policy is now stated: anonymous `s3:GetObject` under `public/*`,
public-access blocks off for that bucket, `s3:ListBucket` to nobody. With a note that an
org-owned bucket beats a personal one, since artifact links outlive their publisher.

A results section reached a public PR whose entire content was hand-typed to look like
terminal output. Three independent things had to hold for that, and all three did.

The gate is model-invoked, so it can be skipped: the publish and the gate ran as two
statements rather than one chain, and the verdict was read after the write.

The hook that fires on the publish call carried a SECOND, narrower copy of the rules —
keyed on verdict tokens — so a comment rendering no verdict satisfied neither copy.
Two rule sets means the weaker one governs whatever falls between them. The hook now
delegates to `attest-gate.sh`: one rule set, invoked by construction rather than by
choice, in the mode the body's markers imply. It fails CLOSED once it has identified a
body it is about to publish — an enforcement point that waves things through when it
cannot find its rules is not one.

And check 5 in `--diligence` had been rewritten as a phrase denylist ("npm pack",
"complete specifier set"), which is precisely the regression its own comment records as
having shipped four times: every property of plaintext is forgeable by whatever emits
the plaintext. It is a medium test again — if the artifact shows a command or a run
result, it owes the reader something fetchable. `/blob/` links are excluded, because a
permalink to a `.json` file satisfied a naive extension test and was the specific reason
the hand-typed section passed.

Comment-update URLs carry the comment id, not the issue's, so check 12 was asking
whether pull #5177261620 was open. Resolved through the API instead.

Four-arm verified: blocks the exact command and body that shipped; ignores `ls`; ignores
a `gh` read with no body write; refuses when the gate is unreachable.

`mms-evidence` named `/attest` as the gate its output passes and shipped no such thing.
Anyone installing the skill got phase 0 as a script and a reference to a command only its
author had, which is half a publish path for everyone else.

Phase 0 is `attest-gate.sh`, already here. Phase 1 is three briefs sent to fresh
instances — frame, coverage, and how it reads to a stranger — written out in
`references/dispatched-passes.md` so dispatch does not depend on commands that live
elsewhere. The check table in `references/phase-0-checks.md` is generated from the gate
rather than retyped, so it cannot drift from what runs.

Two things the port makes explicit that the original left to discipline. The gate must be
the same shell chain as the publish, because running both and reading the verdict after
the write is how a blocked artifact reached a public PR. And softening a check to fit the
case in hand is called out as an anti-pattern: if the new version could be satisfied by
better prose alone, it is no longer the check.

Three copies of this hook were on one machine and the oldest was the one wired into
settings — no `gh api` matcher, no delegation to `attest-gate.sh`. Every publish through
that path went ungated, and nothing noticed, because a gate that blocks nothing looks
exactly like a gate with nothing to block.

`gate-controls.sh` is the thing that would have noticed. Six arms: three publish routes
that must block, three inputs that must pass. It copies the hook somewhere with no sibling
`scripts/` so `_find_gate()` resolves the way it does in production rather than the way it
does in a checkout — the difference matters, and testing the checkout copy is how the
deployed one stayed broken.

Both halves earned their place immediately. The negative arm caught that delegating to
attest-gate ran it over EVERY published body, so an ordinary reply was judged as a failed
validation run; wiring that would have blocked every normal comment. The gate now applies
only to bodies carrying an artifact marker or a verdict line.

And the positive arm caught itself: the enrichment probe was a single sentence, which that
rule correctly ignores, so the arm had been passing because attest-gate blocked the body
for an unrelated reason. A probe that fires for the wrong reason reports a working rule.

The `enrichment` class is ported forward from the older copy, with the constant it depends
on — it existed in the deployed version and in neither newer one.

The six arms prove the script blocks what it should. They say nothing about whether
anything calls it, and those are different questions — a hook that is unwired, or wired
to a path that no longer exists, is indistinguishable from a hook with nothing to block.
One session ran start to finish with every PreToolUse hook inert: 306 certification
markers written, none enforcing anything, and the ritual read as compliance.

The check enumerates config roots rather than trusting `$HOME`. Its first version did
trust it, found one settings file, reported it as "the" wiring and never looked at the
second — because `$HOME` here points at a per-account directory rather than the login
home. That is the same defect one level up, caught only because two configs were known
to exist and one was missing from the output.

It stops short of claiming liveness, and says so: a settings file naming an existing
file is not proof the running session loaded it. Only a command the gate must block,
issued in a session and observed to be blocked, shows that.

This is the bypass. The hook reads the command as text, so `--body-file $DIR/comment.md`
resolves to nothing — and the code called that "can't read it -> nothing to scan" and
allowed the write. Every publish in one long session used a shell variable for the path,
so every one of them went ungated, including a comment whose entire results section was
hand-typed to look like terminal output. That same body, passed by literal path, is blocked.

An unreadable body is not an absent risk. By that point the command is already identified
as an outward-facing write; not knowing what it carries is the reason to stop.

Also rejects a body argument carrying `$` or a backtick — `--body "$(cat f)"` extracts the
literal characters, scans clean, and publishes whatever the shell substitutes afterwards.

The check reads the command ARGUMENT, not the body. A first version scanned body text for
shell metacharacters and blocked every evidence comment ever written, because markdown
inline code is backticks and these artifacts are full of them. Caught by the negative arm,
which is the half of a control matrix that earns its place on days like this.

Three arms added, verified end to end: the command that slipped through minutes earlier is
now stopped by the deployed hook.

This branch adds `hooks/pr-evidence-gate.py` and `hooks/gate-controls.sh`, and on main
neither would have survived: `hooks` is absent from `BUNDLE_DIRS`, so the linter rejects
the directory outright, and absent from the bundle loop in `tools/install`, so nothing
would copy it even past lint. The gate would have merged into the repository and installed
to nobody.

`hooks` now appears in both.

The third change is the one worth keeping. The two lists are a single contract written
twice, in JavaScript and in Bash, and the schema's own comment asked humans to keep them
in sync by hand. `tools/check-bundle-contract.sh` asserts it instead. It builds a fixture
skill carrying one file in every declared bundle directory, runs a dry-run install from a
copy of the installer placed inside that tree, and checks each directory is reported as
copied. It watches behaviour rather than reading the loop, because a regex over the source
is the same class of evidence as the defect: two declarations agreeing with each other and
neither of them being what runs.

Controlled both ways. It passes as committed. Removing `hooks` from the loop makes it name
`hooks` and exit 1; removing three makes it name three.

Also relocates `probes/metametrics-context.test.tsx`, which lint rejected as an unexpected
directory, to `assets/` where a copy-out template belongs, and points the C4 lane at it.
It had been shipping with nothing in the skill naming it, which is the same defect one
layer over: present, and unreachable.

The hook already ran attest-gate.sh in-process on the exact bytes about to be
published, which is a stronger binding than recording a verdict against a
hash — there is no window in which the artifact can be edited after the gate
passes. Two defects meant it often did not run, or ran the wrong gate.

1. _is_evidence_artifact matched `^\*\*Verdict:\*\*`, so where the bold
   stopped decided whether thirteen checks ran:

     **Verdict:** proven     -> all 13 checks
     **Verdict: proven**     -> none

   The same sentence, rendered identically, one of them silently unenforced.
   Now matches a line-leading bolded Verdict however the emphasis falls, still
   anchored so prose mentioning the word does not drag an ordinary reply in.

2. _find_gate ranked $ATTEST_GATE last, behind three default paths. An
   override that loses to a default is not an override: a control run pointing
   it at a stand-in silently exercised the installed gate and reported on that
   instead. This is the failure the skill's own non-negotiable 9 describes —
   an instrument reporting the instruction it was given rather than the effect
   it had — and it was in the resolver for the gate itself.

Verified by substituting a stand-in gate whose exit code is controlled, with
the hook copied to a directory with no sibling scripts/ (as gate-controls.sh
does, for exactly this reason):

                          before   after
  stand-in exits 0        ALLOWED  ALLOWED
  stand-in exits 1        ALLOWED  BLOCKED
  gate path nonexistent   ALLOWED  BLOCKED

gate-controls.sh: all eleven arms behave, including the three negative arms
that catch a gate which has started blocking everything.

…ntmatter mean something

Three controls that were present and inert.

1. evidence-run.yml's determinism check emitted ::warning:: and carried
   continue-on-error, so the step whose own message reads "do not publish
   these numbers" went green and the numbers published. Now ::error:: plus
   exit 1, with continue-on-error removed from that step only. The two runner
   steps keep it, because there the exit code is the verdict and a finding is
   not a failure. Here a difference means the instrument did not return the
   same answer twice, so neither answer is publishable.

   Verified by extracting the step and running it: identical arms exit 0 —
   including the label/log/logs/env fields it deletes by design, which is the
   false positive that taught operators to publish through it — and differing
   arms exit 1.

2. selector-recompute.sh returned 0 for every outcome including
   "VALUE UNSTABLE" and "probe-failed", so a caller gating on the exit code
   saw green on a run whose own artifact says the number is not meaningful.
   The only thing between that and publication was attest-gate happening to
   grep the verdict string out of the prose.

   Now 4 for VALUE UNSTABLE and 5 for probe-failed, with 0 kept for both real
   measurements — a selector that recomputes is a result, not an error. Codes
   documented in the header, and the --help range extended so it shows them.

3. Three of fifty skill.md frontmatters were invalid YAML: `description` was a
   plain scalar containing ": ", which YAML reads as a nested mapping. Any
   tool parsing source frontmatter fails on them; tools/install masked it by
   folding the value to a block scalar on the way out, so the installed copy
   parsed and the source did not.

   Folds evidence and attest to block scalars at rest. The performance skill
   has the same defect but lives on main, so it is fixed in the tools/install
   branch rather than here.

   Round-trip checked: the installed description is byte-identical to the
   source description after folding.

`node --test test/*.test.mjs` 61/61.

… a claim

Broadening the artifact trigger made it fire on text ABOUT the trigger. A pull
request quoting `**Verdict:** proven` to show what the gate matches was
classified as a validation run and asked for the whole envelope, so
documenting the rule became a violation of it. Fenced blocks are stripped
before the trigger is tested.

  real verdict in prose                    -> artifact
  real verdict, bold-wrapped               -> artifact
  verdict ONLY inside a fence              -> not an artifact
  fenced example alongside a real one      -> artifact
  ordinary reply                           -> not an artifact
  explicit VALIDATION_RUN marker           -> artifact

gate-controls.sh 11/11; chaining controls still verified; fabricated-artifact
controls still 3/3 blocked.

Surfaced by the trigger now working: a real published body is blocked by the
CI-restatement rule, which had never run against it because the old trigger
did not match its verdict line. That rule and this package's own direction
disagree, and the conflict is left for the owner rather than resolved here —
see the pull request description.

The rule is that a validation surface does not restate CI status: "tests are
green at head <sha>" hands the reviewer their own Checks tab back and carries
no information. The implementation matched a bare `actions/runs/N`, which is a
different thing — five of its six branches described a CLAIM about CI, and one
described a URL.

So a link to a specific run and job whose log holds the figure being reported
was refused, and that is precisely what evidence-run.yml exists to produce:
"move the measurement to CI, where the run URL is the capture". The package
forbade its own flagship output, and a real published body was blocked by it.

A run link is now a violation only when it carries restatement language with
it. The distinction is whether the sentence asserts a status the Checks tab
already shows, or points at an execution whose output the Checks tab does not.

scripts/ci-citation-controls.py holds the line, with five restatements that
must be caught and three citations that must be allowed. It imports
CI_RESTATEMENT from the hook rather than restating it, because a control that
tests its own copy of a pattern passes forever while the real one drifts.

  before: 3 of 8 misclassified — every citation a false positive
  after:  0 of 8
  the same controls against the old pattern still catch the citations, so the
  control has power rather than passing by construction

gate-controls 11/11; replaying real published bodies now shows zero
regressions, where the CI rule previously blocked one; principle-coverage
unchanged at 3/9; `node --test` 61/61.